Skip to content

docs: RFC for JSONL streaming format#14

Merged
eliot-emp merged 8 commits intoempathic:mainfrom
eliothedeman:docs/rfc-jsonl
Apr 17, 2026
Merged

docs: RFC for JSONL streaming format#14
eliot-emp merged 8 commits intoempathic:mainfrom
eliothedeman:docs/rfc-jsonl

Conversation

@eliothedeman
Copy link
Copy Markdown
Collaborator

Summary

  • Adds docs/RFC-jsonl.md: a draft RFC defining a line-oriented JSONL companion to Toolpath's canonical JSON format.
  • Targets two primary use cases: live agent traces and multi-writer append logs.
  • Round-trips losslessly with canonical JSON; signatures survive stream ↔ seal cycles.
  • Requires one additive schema change (documented in the RFC, not implemented here): optional graph_ref on PathIdentity.

Docs-only. Implementation will follow in a separate PR once the RFC is accepted.

Test plan

  • Review RFC for technical correctness
  • Confirm style matches docs/RFC-correlation.md
  • Confirm scope decisions (atomic steps, one-path-per-file, strict parsing, lossless round-trip)
  • Confirm proposed PathIdentity.graph_ref addition is acceptable

Defines a line-oriented JSONL companion to the canonical Toolpath JSON
format, optimized for incremental persistence of Path documents. The
format supports live agent traces and multi-writer append logs while
round-tripping losslessly with canonical JSON.

Requires one additive schema change: an optional graph_ref field on
PathIdentity so a streamed path can name the graph it belongs to.
- Define .path.json (canonical) and .path.jsonl (streaming) extensions
- Replace .toolpath.jsonl with .path.jsonl in the JSONL RFC
- Add constraint: graph $ref entries must point to sealed .path.json
- Add File Extensions section to the base RFC
Multi-writer append logs are not a realistic use case. Simplify the
motivation, durability, and design rationale sections to focus solely
on the live agent trace use case.
Sealing collapses intermediate PathMeta overwrites and does not
preserve line ordering, so JSONL -> JSON -> JSONL is lossy. The
actual guarantees are:
- JSON -> JSONL -> JSON is lossless: seal(stream(P)) == P
- The sealed form is stable: seal(stream(seal(L))) == seal(L)

Updated the abstract, goals, round-trip section, signatures section,
and strict parsing rationale to reflect this.
The "sealing" and "streaming" abstractions added ceremony without
value — they're just "how to read a JSONL file" and "how to write
a JSONL file." Renamed throughout:

- Sealing Algorithm -> Reading JSONL
- Streaming Algorithm -> Writing JSONL
- seal()/stream() -> read()/write() in round-trip identities
- Updated all references in abstract, goals, signatures,
  compatibility, open questions, and design rationale
- Collapse if-blocks into match guards (collapsible_match)
- Use sort_by_key with Reverse instead of sort_by (unnecessary_sort_by)
Forward compatibility matters more than strict rejection. Older
readers should produce a correct (if incomplete) view of a path
written by a newer producer, rather than refusing to read it.

Unknown lines are skipped with a warning. Version bumps remain
available for truly incompatible structural changes.
Copy link
Copy Markdown
Contributor

@akesling akesling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eliot-emp eliot-emp merged commit 5c5a0d0 into empathic:main Apr 17, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants